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Statistical Self-Calibrating Detection and Removal of Blemishes in 
Digital Images 

BACKGROUND 

5 1. Field of the Invention 

This invention related to digital photography and in particular, automated means of 
removing blemish artifacts from images captured and digitized on a digital process. 

2, Description of the Related Art 

Many problems are caused by dust in particular and blemishes in general on imaging 
10 devices in general and digital imaging devices in particular. In the past, two distinct forms of 
image processing included providing means to detect and locate dust, scratches or similar 
defects and providing means to remedy the distortions caused by the defects to an image. It 
is desired to have an advantageous system can automatically detect and correct for the effects 
of dust, scratches and other optical blemishes. 
15 Dust has been a problem in scanning devices for a long time. Various aspects of the 

scanning process, enhanced by image processing techniques where appropriate, have been 
employed to provide means for the detection of dust or defects relating to document or 
image/film scanners. These devices form an image by moving a ID sensor pixel array across 
a document platen, or in some cases, the document platen, with document/image is moved 
20 under the sensor array. The physics and implementation of these devices differ significantly 
from those of a field-based sensor or camera-style device. It is desired particularly to have 
dust and/or blemish detection and correction techniques for field-based or camera-style 
acquisition devices. 

Image correction has been studied in relation to display devices, output apparatuses 
25 such as printers, and digital sensors. Image correction of dust artifacts can be used to recreate 
missing data, also referred to as in-painting or restoration, or undoing degradation of data, 
which still remains in the image, also referred to as image enhancement. It is desired to have 
a system including a digital camera and an external device or apparatus that can facilitate 
defect detection and/or correction involving sophisticated and automated computerized 
30 programming techniques. 



SUMMARY OF THE INVENTION 
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The present invention provides a method of automatically correcting dust artifact 
regions within images acquired by a system including a digital acquisition device according 
to claim 1. Preferred embodiments include acquiring multiple original digital images with the 
digital acquisition device. Probabilities that certain pixels correspond to dust artifact regions 
5 within the images are determined based at least in part on a comparison of suspected dust 
artifact regions within two or more of the images. Probable dust artifact regions are 
associated with one or more values of one or more extracted parameters relating to the lens 
assembly of the digital acquisition device when the images were acquired. A statistical dust 
map is formed including mapped dust regions based on the dust artifact determining and 
10 associating. Pixels corresponding to correlated dust artifact regions are corrected within 
further digitally-acquired images based on the associated statistical dust map. 

The dust artifact probabilities may be determined by statistically combining multiple 
individual probabilities based on each of the regions within two or more images. 
Probabilities may alternatively be determined that certain pixels within the image correspond 
15 to regions free of dust based at least in part on a comparison of suspected dust artifact regions 
within one or more of images. Certain suspected dust artifact regions may be eliminated 
based on probabilities that certain pixels correspond to regions free of dust, and/or as having 
a determined probability below a first threshold value that the suspected region is in fact dust 
artifact. 

20 Certain further dust artifact regions may be judged as having a probability above the 

threshold value, such as to be subject to further probability determining including comparison 
with further acquired images prior to judging whether each further dust artifact region will be 
subject to the eliminating operation. There may be only a single threshold such that above 
the threshold, the suspected region will be judged to be subject to a dust correction operation, 
25 or a second threshold above the first threshold may be applied for that purpose, and there may 
be further thresholds defined within the software. 

The probability determining operation may include weighting suspected dust artifact 
regions according to one or more predetermined probability weighting assessment conditions. 
These conditions may include size, shape, brightness or opacity, feathering, or peripheral 
30 smoothness of said suspected dust artifact regions, or a degree of similarity in size, shape, 
brightness, opacity or location with one or more suspected dust artifact regions in one or 
more other images, or combinations thereof. 

The one or more extracted parameters may include aperture size, F-number, 
magnification, lens type, focal length of the digital acquisition device, or combinations 
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thereof. The one or more extracted parameters may be calculated empirically from 
comparison of one or more dust artifact regions within multiple original digital images with 
the digital acquisition device. 

The further digitally-acquired images may include different images than the originally 
acquired images, and may further include one or more of the same original images. The 
different images may be acquired with different values of one or more extracted parameters, 
they may be acquired of a different object or objects or of a different scene or scenes. 

The determining and associating operations may be repeated for the further digitally- 
acquired images, and the statistical dust map including mapped dust regions may be updated 
based on the additional dust artifact determining and associating. The determining and 
associating may also be repeated for additional images acquired with the digital camera, and 
the statistical dust map including the mapped dust regions may be updated based on the 
additional dust artifact determining and associating. Pixels may be corrected corresponding 
to correlated dust artifact regions within the additional images based on the updated, 
associated statistical dust map. One or more of the further and original images may be 
updated based on the updating of the associated statistical dust map. The updating for the 
further and original images may be limited to updates that do not include appearance of new 
dust or movement of existing dust, and/or to updates that include previously determined dust 
artifact regions. 

A version description may be created of changes in the statistical dust map. One or 
more of the further and original images may be updated based on the updating of the 
associated statistical dust map based on the version description. The version may be based on 
a chronological time stamp, and/or replacement of a lens. Version information may include a 
change of probability in the statistical dust map that certain pixels correspond to dust artifact 
regions. The version information may include one or more parameters including a change in 
dust location, change in dust position, appearance of a new dust region, and/or a 
disappearance of an existing dust region. A determination may be made as to whether a dust 
map is to be replaced based on determining that sufficient disparity exists based on an 
amount and/or quality of changes in the statistical dust map. 

The image correction method may be automatically performed within a digital camera 
that includes an optical system, a sensor array, processing electronics and a memory. The 
image correction method may also be performed at least in part within an external processing 
device that couples with a digital camera that includes the optical system and the sensor array 
to form a digital image acquisition and processing system that also includes the processing 



4 

electronics and the memory. The programming instructions may be stored on a memory 
within the external device which performs the image correction method. The digital 
acquisition device may capture images from film images, and the digital acquisition device 
may include a digital camera. 
5 The dust artifact probability determining may include dynamically updating the 

probabilities based on comparisons with suspected equivalent dust artifact regions within the 
further digitally-acquired images. The determining of probabilities is further based on a pixel 
analysis of the suspected dust artifact regions in view of predetermined characteristics 
indicative of the presence of a dust artifact region. The dust artifact probability determining 
10 may include determining probabilities that certain pixels correspond to dust artifact regions 
within acquired images based at least in part on a comparison of suspected dust artifact 
regions within two or more digitally-acquired images, or on a pixel analysis of the suspected 
dust artifact regions in view of predetermined characteristics indicative of the presence of a 
dust artifact region, or both. 
15 The probability determining may further include statistically combining probabilities 

based on comparisons of inner or shadow regions and of aura regions. The probability 
determining with respect to a shadow region of a dust artifact may be based on an extracted 
parameter-dependent shadow region analysis, wherein the shadow region analysis presumes 
that certain regions on a sensor of the digital image acquisition device are fully obscured by 
20 the dust. The probability determining with respect to an aura region of the dust artifact may 
be based on an extracted parameter-dependent aura region analysis, wherein the aura region 
analysis presumes that certain regions on a sensor of the digital image acquisition device are 
partially obscured by the dust. The aura region analysis may include calculating effects of 
differences in values of the one or more extracted parameters in different images of dust 
25 artifact illumination, shape, position, reflection or transmission properties, distance of dust to 
the sensor, aperture, exit pupil, or focal length, or combinations thereof. The different images 
may be acquired with different values of the one or more extracted parameters, and the 
different images may be acquired of different objects. 

The method may further include validating whether a further digitally-acquired image 
30 has non-contradicting data that the probability that certain pixels correspond to dust artifact 
regions within the image prior to correcting pixels corresponding to correlated dust artifact 
regions within the images based on the associated statistical dust map. The user may be 
instructed to create a new dust map if the validating determines a noticeable change in the 
dust map. The shadow region analysis may include calculating effects of differences in 
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values of the one or more extracted parameters in different images on dust artifact 
illumination, shape, position, reflection or transmission properties, distance of dust to the 
sensor, aperture, exit pupil, or focal length, or combinations thereof. 

Suspected dust artifact regions of the further digitally images may include inner or 
5 shadow regions and aura regions. Focal length extracted parameters may be calculated 
empirically from comparison of the transposition of inner or shadow regions of dust artifact 
regions within multiple original digital images acquired with the digital acquisition device. 
Aperture extracted parameters may be calculated empirically from comparison of fall off of 
aura regions of the dust artifact regions within the multiple original digital images acquired 
10 with the digital acquisition device. The correcting may include a first correcting of the aura 
regions and a second correcting of the inner regions, in either order. 

The method may be performed on raw image data as captured by a camera sensor. 
The method may be performed on a processed image after being converted from raw format 
to a known red, green, blue representation. The correcting operation may include replacing 
15 pixels within one or more digitally-acquired images with new pixels. The correcting may 
include enhancing values of pixels within one or more digitally-acquired images. Correcting 
instructions may be kept in an external location, such as an image header, to the image data. 

The dust artifact probability determining operation may include loading the statistical 
dust map, loading extracted parameter information of a present image, performing 
20 calculations within the statistical dust map having extracted parameter variable-dependencies, 
and comparing dust artifact detection data with the extracted parameter dependent statistical 
dust map data. The extracted parameter information may include values of aperture size, 
focal length and/or lens type information. The dust artifact probability determining operation 
may also include loading the statistical dust map, loading extracted parameter information of 
25 a present image, performing a calculation for relating the statistical dust map with the present 
image according to a selected value of an extracted parameter which is otherwise 
uncorrected between the present image and the dust map, and comparing dust artifact 
detection data with the now correlated statistical dust map data. 

Suspected dust artifact regions of at least two images may include inner or shadow 
30 regions and aura regions. The comparison may include a comparison of the inner regions and 
a different comparison of the aura regions, in either order. The dust artifact regions may 
include an aura region partially obscured by dust and a shadow region substantially obscured 
by dust inside the aura region. 
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The determining with respect to a shadow region may be based on an extracted 
parameter dependent shadow region analysis, wherein the shadow region analysis presumes 
that certain regions on a sensor of the digital image acquisition device are substantially 
obscured by said dust. The shadow region analysis may include calculating effects of 
5 differences in values of the one or more extracted parameters in different images on dust 
artifact illumination, shape, position, reflection or transmission properties, distance of dust to 
the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

The determining with respect to an aura region may be based on an extracted 
parameter dependent aura region analysis, wherein the aura region analysis presumes that 
10 certain regions on a sensor of the digital image acquisition device are partially obscured by 
said dust. The aura region analysis may include calculating effects of differences in values of 
the one or more extracted parameters in different images on dust artifact illumination, shape, 
position, reflection or transmission properties, distance of dust to the sensor, aperture, exit 
pupil, or focal length, or combinations thereof. 
15 The correcting operation may include in-painting or restoration, or both. The in- 

painting correcting may be applied to the shadow region. This in-painting may include 
determining and applying shadow region correction spectral information based on spectral 
information obtained from pixels outside the shadow region. The restoration correcting may 
be applied to the aura region. The restoration may include determining and applying aura 
20 region correction spectral information based on spectral information obtained from pixels 
within the aura region. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a main workflow of a dust removal process in accordance with a 
25 preferred embodiment. 

Figure 2A illustrates the creation of a dust map. 

Figure 2B illustrates an alternative embodiment of the creation of a binary dust map. 
Figure 3 outlines a correlation of a dust map to image shooting parameters. 
Figure 4A illustrates a procedure for removing dust from images in accordance with a 
30 preferred embodiment. 

Figure 4B illustrates reconstruction in-painting of dust regions. 

Figure 4C illustrates the occurrence of dust in a picture with high edge information. 

Figures 4D-4E illustrate the numerical gradient in an image. 
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Figure 4F illustrates a spiral procedure of in-painting from the periphery inwards. 

Figure 5 illustrates a quality control method of cheeking the dust map. 

Figures 6A-6F represents an optical geometry of a lens that may be used in 
accordance with a preferred embodiment: 

Figures 6A, 6B and 6C represent three lenses with the same, fixed, focal distance, 
with the same focal number, but with different constructions. 

Figure 6D illustrates the concept of an exit pupil and the distance to it. 

Figure 6E illustrates the intersections of the principal ray with window and image 
plane, which is preferably approximately the sensor plane. 

Figure 6F illustrates the shift of one dust spec in comparison to a shift of another dust 
spec in an image. 

Figures 7A-7G generally illustrate effects of dust on the creation of an image using a 
known optical system: 

Figure 7A illustrates an influence of a dust particle located on the input surface of the 
window on the beam converging towards the imaging point. 

Figure 7B illustrates a side view of the rays as obscured by dust as a function of the 
aperture. 

Figure 7C illustrates a frontal projection of the rays as obscured by a dust as a 
function of the aperture. 

Figure 7D and Figure 7E illustrate a power distribution map of the same dust spec as 
manifested in different f-stops. 

Figure 7F illustrates an effect of the change in focal length on the area obscured by 

the dust. 

Figure 7G illustrates the direction of movement of dust as a function of the change in 
focal length in relation to the center of the optical axis. 

Figure 8 illustrates an adjustment of the dust map based on aperture. 
Figure 9 illustrates an adjustment of a dust map based on focal length. 

Figure 10 illustrates a process of estimating based on empirical data the values of the 
parameters in the formulae that defines the change in dust as a function of change in focal 
length. 

Figure 1 1 illustrates a process of estimating based on empirical data the values of the 
parameters in the formulae that defines the change in dust as a function of change in aperture. 
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Figure 12 illustrates a further process of dust analysis in accordance with another 
embodiment. 

Figure 13 illustrates how a portion of the dust map is created/updated based on the 
process described in Figure 2(a). 

5 Figure 14 iJJustrates how a portion of the dust map is created/updated based on the 

alternative process described in Figure 2(b). 

SOME DEFINITIONS 

Dust specs: The preferred embodiment takes advantage of the fact that many images 
10 may have repetitive manifestation of the same defects such as dust, dead-pixels, burnt pixels, 
scratches etc. Collectively, all possible defects of this nature are referred to in this application 
as dust-specs or dust defects. The effects of those dust specs on digital images are referred to 
herein as dust artifacts. 

Acquisition device: the acquisition device may be a multi-functional electronic 
15 appliance wherein one of the major functional capabilities of said appliance is that of a digital 
camera. Examples can include a digital camera, a hand held computer with an imaging 
sensor, a scanner, a hand-set phone, or another digital device with built in optics capable of 
acquiring images. Acquisition devices can also include film scanners with area-capture 
CCDs, as contrasted, e.g., with a line scan mechanism. 
20 D-SLR: Digital Single Lens Reflex Camera. A digital camera where the viewfinder is 

receiving the image from the same optical system as the sensor does. Many D-SLRs, as for 
SLR cameras, have the capability to interchange their lenses, this exposing the inner regions 
of the camera to dust. 

25 A few parameters are defined as part of the process: 



N Number of images in a collection. 

HIDP Number of images for occurrence to be high dust probability 

HSDP Number of recurring specs for label a region to be high dust 

probability 

30 p(Hdp) Probability threshold for high confidence of a dust spec . 

N dp Number of images to determine that a region is not dust 

p(Ndp) Probability threshold to determine that a region is not a dust region. 
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HTDP <= HSDP 

is a generic image 

pixel in location x horizontal, y vertical of Image I 
is a continuous tone, or a statistical representation of a dust map 
is a binary dust map created from some thresholding of dM. 

MATHEMATICAL MODELING OF THE OPTICAL SYSTEM 
Prior to understanding the preferred embodiments described herein, it is helpful to 
understand the mathematical modeling of the camera optical system. With this modeling, the 
preferred embodiments may advantageously utilize a single dust map for dust detection 
and/or correction techniques rather than creating a dust map for each instance of dust artifact 
having its own shape and opacity depending on extracted parameters relating to the imaging 
acquisition process. With an ability to model the optical system and its variability, a single 
map may suffice for each lens or for multiple lenses, and for multiple focal lengths, multiple 
apertures, and/or other extracted parameters as described in more detail below. 

In order to study the shadow of an object on a plane, it is helpful to consider the 
following: 

- the illumination of the object (the spectral and coherent characteristics of 
the light) 

- the shape of the object (including its micro-geometry) 

- the reflection and transmission properties of the object 

- the relative position of the object and the plane 

The case of interest for defining the model of dust as it appears on the sensor, is that 
of an object close to the image plane of a lens, which is a complex optical system, with 
natural light illumination. The object shape and its reflection and transmission properties are 
practically impossible to model because specific information on the dust particle(s) is not 
available. However, because the dust particles are small, it is highly probable they will have 
the same reflection and transmission properties on their surfaces. The distance between the 
dust and the sensor (which lies in the image plane) is small and is in the order of magnitude 
of a fraction of a millimeter. 

Some definitions are now provided (see figures 6E and 6F): 



Most likely 
I 

Kx,y) 

dM 

pM 
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Pe- 


exit pupil position, the distance between the sensor, and the Exit Pupil 


tw - 


thickness of the window, distance of dust to the image plane 


r- 


focal length of the objective 


f/#- 


focal number 


(u.v) - 


denote the coordinate SVStem of the innnf surfarp hnvino iK nricnn in rh/* 

intersection point with the lens optical axis. 


fx v)- 


ihe dust nosition on the imape nlane in the mnrriinafp v:vv:tpm in th*» imnoA T>ia™<» 

1.1 ■ w uuji puoiuuu vii niv* iinut^v pitiuv^ 111 lllv* L-UUIUIllulC oYMCIIl HI IJ1C lllJcltiC plane 

with the oriein in the intersection noint with the lenc nntiral nyic 


l(x v)- 


De the value of the irradiance in fhp imaop nnint \;) 

1 1 1 » UlUb' V/l lllv illUUlUIivw ill 11 1 till fliit/ L/V/Ull l^t V/ 


D - 


is the exit pupil diameter 


ho 


s the distance of the objects on the dust plane to the center of the optical path 


h i 


s the distance from the object to the center of the optical axis 




s the distance of dust spec k on the dust plane to the center of the optical path 



Table 1 . Parameters in mathematical formulation of the Optical system. 



Figure 6A, 6B and 6C represent three lenses with the same, fixed, focal distance, with 
5 the same focal number, but with different constructions. The first construction, in figure 6A is 
the most common. The second in figure 6B is specific for the lenses used in metrology. This 
type of lens is called telecentric in the image space. The third construction in figure 6C is 
rarely found in optical systems, but not impossible 

Figure 6D illustrates the concept of an exit pupil and the distance to it. The exit pupil 
10 distance, 642, is the parameter that defines this property, and represents the distance from the 
secondary principal plane, 646, of the lens and the intersection of the axis of the light cone 
and optical axis of the lens. The focal distance 644 represents the distance from the secondary 
principal plane to the image plane. 

In the case of the second type lens, as defined in Figure 6B, the exit pupil position is 
15 at infinity. In the first and third cases, figures 6A and 6C respectively, the intersections of the 
light cone with the dust plane are ellipses. In the case of the second lens type, Figure 6B, the 
intersection of the light cone with the dust plane is a circle. 

In the case of zoom lenses, the exit pupil position can be constant (in a very few 
cases), or can vary significantly, depending on the construction of the lens. 
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This information about the exit pupil is usually not public because, in general, it is not 
useful for the common user of photographic or TV lenses. However, this information can be 
readily determined with some measurements on an optical bench. Alternatively, this 
information can be achieved based on an analysis of dust in the image. 

Figure 6E illustrates the intersections of a principal ray with the window and image 
plane, which is also the sensor plane. This figure illustrates the various parameters as defined 
in Table 1. 

Figure 7A describes the influence of a dust particle (obscuration) located on the input 
surface of the window (obscuration plane), on the beam converging towards the imaging 
point. 

Figure 7B illustrates the side view of the rays as obscured by a dust as a function of 
the aperture. Figure 7G illustrates the frontal projection of Figure 7B, of the rays as obscured 
by a dust as a function of the aperture. Figure 7D and Figure 7E illustrates a power 
distribution map of the same dust spec as manifested in different f-stops, namely a relative 
open aperture f-9 for Figure 7D and a closed aperture, namely f-22 and higher, in Figure 7E. 
One can see that the dust spot as recorded with the closed aperture, is much sharper and 
crisper. 

The dust particle is situated in the window plane, and totally absorbing or reflecting 
the portion of the beam (S2) that normally reaches the image point (P). Let I(x,y) be the 
value of the irradiance in the image point (x,y) in the absence of the dust particle. Assuming 
that the energy in the exit pupil of the optical system is uniformly distributed, then, in the 
presence of the dust particle, the I(x,y) will be reduced proportionally with the ratio between 
the area of the intersection of the light beam (SI) with the clean zone of the window, and the 
whole area of the same intersection (Sc). 

Remember that Sc is a function of the lens f-number (f/#) and window thickness t w 
So, the value of the real irradiance in the presence of dust will be 

/■(^)=/(^|l (1) 

or, by taking into account that SI = Sc - S2 and S2 = Area of intersection between the dust 
particle and the illuminated area: 

/•U,)=/U^)[l-f] = /(,,y)[i-^ 

(l-a) 
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Because the dust panicle is small, the next assumption can be made: 

For all the image points affected by a specific dust particle, the area Sc remains 
constant. In the case of the telecentric illumination, this assumption holds true for all the 
image points . With the assumption above, we can correlate the dust geometry, position and 
shadow. 

First of all, we now study the distribution of the irradiance on the input plane of the 
window. The intersection of the conic beam with this plane is generally an ellipse. The major 
axis of this ellipse is along the intersection of the plane mentioned above and the plane 
determined by the image point and the optical axis of the lens. 

The minor axis is 



whereas the major axis is 



a=t„ 
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where D is the exit pupil diameter , so 
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20 In order to reach a complete formula of the irradiance distribution we 

now stress the assumption that for all image points affected by a chosen dust particle, "a" will 
not vary significantly. 

Let (uo ,vo) be the point of intersection of the principal ray with the plane mentioned 
above. Thus, the characteristic function of the illuminated zone will be 
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Let D(u ,v) be the characteristic function of the dust/Thus, 
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which, using the equation II, yields 
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The terms (7 +tJP t )x and (I+tJP e )y determine the "movement" of the dust shadow with the 
10 change of the focal distance and, implicitly, of the Pe, as explained later on in equation 1 1. 
In the case of the telecentric illumination 

Pe is infinite , 
- uO = x and vO=y. 

the ellipse becomes a circle 
15 Qualitatively, as described quantitatively in equation 8, the fall off, or the diminished 

effect of the dust on an image varies as follow: The dust is becoming less significant as the 
aperture becomes larger, or the f-stop is smaller and the pixels are inversely affected based on 
the distance of the pixel to the periphery of the dust. 

Figure 7F illustrates the adaptation between the dust map and the image based on the 
20 focal length of the lens. In this figure one can qualitatively see that the area covered by the 
dust spec will be shifted as a function of the focal length. Also, information about the spatial 
distribution and/or orientation of dust particles within the dust map may be used to assist with 
determining probabilities that regions are either dust or non-dust. As an example, one could 
expect that a global average of the orientation of elongated dust particles would average to 
25 zero. However, at a more localized level, charged and/or magnetized dust particles can tend 
to align with similar orientations due to local dielectric forces. This effect would be partly 
dependent on the size, overall shape distribution (round or more elongated) and the overall 
distribution (uniform or clumped) of the dust regions in the dust map. This can tend to be 
particularly helpful when a decision is to be made as to whether to eliminate certain regions. 
30 As an example, if most of the dust is determined to be aligned in a radial direction relative to 
the center of the sensor area for a particular camera, it may be further determined, based on 
this statistical information, that particles in a concentric alignment are less likely to be dust. 
Given the definitions above, a more quantitative model can be deducted. 
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because the ratio is small. Note that Pe has in certain common cases a negative value. 

Next let (u,v) denote the coordinate system of the input surface, having its origin in 
the intersection point with the lens optical axis. Similarly, let (x,y) be the coordinate system 
in the image plane with the origin in the intersection point with the lens optical axis. 

Thus, if the principal ray of the beam intersects the input window plane in the point 
(w,v), the point in the image plane on the same principal ray will be the inverse transformation 
of (10) 
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with the same assumption of a small tw/Pe ratio will be 
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Figure 7G depicts how points will shift as a function of the proximity to the optical 
center. Basically the farther the dust is from the center, the larger the displacement is. 
However, the direction and size of the displacements can be estimated. 

There are several other parameters that are not generally available. The distance, tw, 
of the dust to the image plane is typically fixed for a particular model of camera and can be 
determined from a single measurement. The exit pupil distance may vary depending on the 
focal length of the lens. More particularly, for a zoom lens, the exit pupil distance may not be 
fixed, and can vary up to a telecentric mode where the distance is infinity. 

Such information can be empirically predetermined by making an analysis of images 
taken by a known camera and lens, as illustrated in Figure 6F and as explained below. In this 
case, a determination of the effect of a single dust spec on each of a series of image can be 
used to calculate the effect of any other dust spec on any other image, given some additional 
knowledge relating to the acquisition parameters for the latter image, in particular parameters 
relating to the optical sub-systems of the camera and the attached lens. The most useful 
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determination is of the spatial shifting of dust specs between images but other determinations 
relating to the geometric and optical properties of a dust spec can be made from the foregoing 
optical model. As illustrative examples of geometric properties we cite the relative size of a 
dust spec, and distortions in the geometric shape of a dust spec arising from lens aberrations 
(i.e. non-circular lens symmetries). Illustrative examples of optical properties include the 
opacity, induced color shift and sharpness of the dust spec. In Figure 6F an exemplary zoom 
lensof 70mm-21 0mm is illustrated: 

- Pe-70 is the exit pupil position for the lens at 70mm, 

- Pe-21 0 is the exit pupil location when the lens is at its highest enlargement 
210mm. 

- k, is known dust 

- m is hypothetical dust 

- h k is h for a specific dust particle k. 

Knowledge of the spatial shift of dust image, preferably near to one of the comers of the 
image, can be determined from equation (9) 



h=h 0 



(9) 



Now this can be rewritten for a specific dust particle (k) in order to emphasize the 
dependence on the focal distance, 



h k (f)=h 0k 



I p.if)) 



(9-a) 



or 



1 y—r independent from k fo.M 
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So, if we know the evolution of the image of the "k" dust particle we can find the evolution 
of the "m" dust image 

K,{f) = ^Af) (9-c) 

"OA 

A remaining difficulty with this formula is that we don't know the positions of the dust 
particles. 

Writing the upper formula for a given focal distance , say f 0 ,we have 
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K k ' Mfo) (9 ' d) 

So, finally 

K(f)-^U) < 9 -e, 

This relationship does not require a knowledge of the exit pupi] dependence, 
5 the thickness of the window and the spatial location of the dust. Figure 7G, illustrates this 
phenomena visually. Based on the above formulae, and in particular 9-e and 1 1, one can see 
that the dust movement is not constant, as a function of the focal length change, but is 
dependent on the distance of the dust from the center of the optical path. 

Qualitatively, the farther the dust is from the center, the larger the 
10 displacement will be. In other words, dust close to the periphery of the image will exhibit 
significantly larger movement (and geometric and size distortions) than dust which maps onto 
the center of the image. 

Referring to figure 7G which describes the movement of two hypothetical 
dust spots, 790 and 794 having a different distance, 792 and 796 respectively, to the center of 
15 the optical path 780. The movement of each pixel is always on the line between the pixel and 
the center of the optical path. This is based on the assumption of a linear relationship as 
described by equation 11. Thus the movement of the dust edge of dust spec 794 is along the 
lines depicted in block 799. Based on assumed coefficients for equation 11, the dust spec 

moves by a fixed ratio 1+ p , which can be positive or negative, which means movement 

c 

20 towards the center or out of the center. In this figure, dust spec 794 will move to become 795 
while dust spec 790 will move to become 791. Because the distance of dust 794 to the center 
depicted by 796 is larger than the corresponding distance depicted by block 792, the 
movement of the dust 794 will be larger. 

Alternatively, given formula (11) and given an image where the dust can be 

25 detected, one can calculate Pe as follows (if tw is known): 

* y i K 
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In summary of the mathematical model of the optical system for dust, in the 
general case, the way the image is affected by the dust depends on: 

- exit pupil position Pe 

- thickness of the window tw 
5 - focal length of the objective f 

- focal number f/# 

- the dust position on the image plane (x,y) 

This can be calculated by knowing the optical system's components: 
a) the window thickness (tw), 
10 b) the function Pc(f), and 

c) the coordinates (u,v) of the dust if we want to determine the dust image 
position on the image plane 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



Several examples of preferred embodiments of the main invention will now be 
described. The main workflow of detecting and removing the dust from an image is 
illustrated in Figure 1. In general, this particular preferred embodiment is directed at 
removing dust from a collection of images acquired using the same image acquisition device. 

20 Specifically, a user may acquire a picture on the digital camera (as illustrated in Block 101). 
Alternatively (102), a user may open a single image on a external device such as a personal 
computer, open (103) a folder of images on an external device or open a collection of images 
on a digital printing device (104). 

The preferred embodiment then extracts the shooting parameters ( 1 20). Such 

25 parameters include, but not limited to data about: Camera name, Lens brand, lens type, focal 
length at acquisition, aperture range, aperture at acquisition. In addition, some parameter, in 
particular from those relating to the lens and the camera may be also stored within the 
acquisition device or the processing device, or subsystems thereof. Such information may 
include parameters such as exit pupil, exit pupil distance from the lens, or distance of dust 

30 from the sensor plane (imaging plane) for the camera. An exemplary table with such data is 
shown below: 



Field Example of daia 



Category 
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jLens Manufacture! 


Nikon 


lens 


i 

Lens Type 


AF24mm-45mm f2.8-f3.5 


lens 


Focal Length 


38mm 


Acquisition data 


Aperture 


1-16 


Acquisition data 


Dust distance 


0. J 56mm 


Camera data 


Exit pupil 


1 9mm 


Lens data 


Exit pupil distance 


230mm 


Lens data 



Table 2 Extracted Parameters 



The dust map may also include meta-data that contain additional parameters or 
acquisition data which can additionally supplement or partially substitiute for said list of 
extracted parameters. A discussion of meta-data as it relates to image acquisition is found in 
5 more detail at PCT patent application no. PCT/EP2004/008706, and is hereby incorporated 
by reference. 

In the case that the system deals with multiple images (as defined in blocks 103, and 
104), the algorithm describes a loop operation on all images (110). The first step is to open a 
dust map (130). Jf none exists (132) the system will create a new dust map (200) as further 
10 described in Figures 2. In the case that the system has more than one dust map (130), which 
may occur, for example, when a camera has multiple exchangeable lenses, the software will 
attempt to correlate one of the maps (300) to the image. This process is further illustrated in 
Figure 3. In particular this correlation refers to the adjustment of the shooting conditions to 
some accepted standard. The most significant acquisition parameters are the aperture and the 
15 focal length. The correlation process is interchangeable and may be achieved either by 
adjusting the image to the map or, conversely, adjusting the map to the image. In some 
extreme cases it may be advantageous for both acquired image and dust map to be adjusted to 
some common ground. This may occur, for example, when the map is calculated based on an 
aperture that the lens does not reach or based on a different lens than the one used with a 
20 different optical configuration. Alternatively, such as in the case of a new lens, this process 
(300) as called by (130), may be used to adjust the map onto a new map, or to merge two 
preexisting maps into a single one. From that stage onwards the system may continue with a 
single map. 

If no dust map corresponds with the image (140), a new dust map is created (200). 
25 When a dust map does correspond (140) to the image, the preferred embodiment checks if the 
dust specs as defined in the dust map are of high enough confidence levels to be determined 
as dust regions (150). The determining and calculating of said confidence level for the dust 
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map in general and for an individual dust spec, is further discussed in Figures 2 and Figure 3. 
If the confidence level is low, the image is added to the updating of the dust map (200). If 
after the image is added, the confidence level is increased sufficiently (152) the software 
continues to the dust removal process (160). Otherwise, the software progresses to the next 
5 image (170). 

For example, a dust map is considered valid only after a minimum number of images, 
say N, are analyzed, and a dust spec is considered valid after a co-located dust region is 
detected in a second minimum number of images, say M. Note that M< N. In an example 
where N=6 and M=3, then after analyzing 4 images, the software may have determined and 

10 confirmed the existence of a substantial number of dust specs and updated the dust map 
accordingly (200), but it must continue to analyze additional images, 2 more in this example, 
in order to validate the dust map, and allow the software to proceed to the correction (160) 
stage. Prior to achieving said validation of the dust map no correction will be done. However, 
images can be corrected retrospectively after the dust map has been validated. Thus even 

] 5 images that were originally acquired prior to validation of the dust map may be corrected. 

Referring to the dust detection and correction process (160). This process may be 
executed as a loop on every dust spec in the dust map, with a stage of detection and 
correction (400, followed by 500 and 160). Alternatively, the process can be implemented 
where all dust specs are detected first (162) and then all dust specs are corrected (164). The 

20 decision as to the sequence of operations varies based on the requirements of a particular 
embodiment of the invention. As part of the detection process, the software also performs a 
self testing (500) where each new image introduced is compared to the dust map. This 
process is further defined in Figure 5. The importance of this stage for each dust spec and for 
the dust map, is that in this manner, if the dust situation changes, such as a single spec 

25 moving around or the camera being serviced, the software will detect the change and 
determine that re-validation of the dust map is required. Note that the re-validation process 
can be performed locally on individual dust specs, or alternatively on the entire dust map, or 
in certain circumstances through a combination of both local and global validation. 

Referring to Figure 2A where the Dust map creation and updating is defined: This 

30 process can receive a collection of images as defined by Figure 1 blocks 103 and 104, or one 
image at a time, as defined by Figure 1, blocks 101 and 102. 

If the function is called with a single image (240-SINGLE IMAGE) the image is 
directly provided to the calculations (270). If multiple images are provided (240 MULTIPLE 
IMAGES), then an initial step is to define if there are more than enough images for defining 
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the map (220). This step is designed to optimize the creation process for a dust map in case 
there are a large number of images. 

The sequence of the images that are to be referenced is based on the original 
collection of N images as defined in Figure I blocks 103 or 104. The sequence of images is 
based on a few criteria such as: giving more weight to the most recently acquired images, and 
if images are shot in a relatively small time frame, the sequence in which images are 
processed in order to create the dust map is determined so as to try and minimize 
repetitiveness between similar images that may have been taken of the same object with little 
movement. The sequence will not be limited to the number of images (HIDP) because it may 
well be that some regions will not have enough data in them to evaluate and validate dust 
specs within said region. This may happen in cases where part of the image is very dark in 
some of the images. 

Alternatively, the parameters, including metadata, as extracted from the images will 
determine the sequence and the number of dust maps that are to be calculated. For example if 
15 a folder contains N=30 images, where 15 were taken with one camera and 15 with another, 
the sampling step (230) may create two map sets. 

Another criteria for creating a new set or checking for new dust is the type of lens. If a 
lens is changed, it means that the CCD-cavity was potentially exposed to new dust. This may 
trigger a new revalidation process. It may also indicate that the camera was serviced, or that 
20 the photographer cleaned the camera. Of course, if there is a parameter that defines when a 
camera was serviced, this will trigger the creation of a new dust map. 

The next loop (270-271) defines the marking of each region and the addition of the 
region to the dust map if it is not already there. There are three type of regions in an image. 
The first are regions with sufficient information to determine whether they are of dust nature. 
25 As an illustrative example, small dark regions surrounded by a light and relatively large and 
homogenous background area. Other criteria may include regions with a relatively low color 
saturation. The second group are regions that are definitely non-dust. Such regions are for 
example all clear, or of high color saturation. Other regions are inconclusive such as very 
dark segments of an image. In this latter case, it will be hard to detect the dust even if it was 
30 part of the image. Alternatively when looking for over exposed or "dead pixels" the criteria 
may be reversed, if the pixels appear as a white spec in the image. 

These criteria may be also be a function of the acquisition parameter. For example an 
image with a open aperture may all be marked as inconclusive for the determination of dust, 
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because fhe dust may not be focussed sufficiently to appear in determinable form on the 
image. 

Regions that are potentially dust are marked (292) and then added to the dust mask 
(294). The addition process may include the creation of a new dust spec region on the map, or 
5 the modification of the probability function, or the confidence level, or the size or geometric 
shape, or location of the centre of an existing region. Regions that are most likely non-dust 
are marked (282) and then added to the dust mask (284). The addition and/or modification of 
said regions needs to be normalized to the shooting conditions as defined by the dust map 
(300), if this step was not performed prior to entering this function, as optionally defined in 
10 Figure 1. This loop continues over all regions of the image (271). Alternatively (272), each 
region is compared (500) with the dust map to see if any cases exist where the monotonicity 
is broken, i.e. a region that was of high probability to be dust is now non dust. 

An illustrative example of how the dust map creation/updating process of Fig 2(a) can 
create new dust or non-dust pixels in the dust map is given in Figs 13(a)-(c). Starting with the 
15 dust map after the image T* has been processed, Fig 13(a), the diagrams show how the 
increasing of the values of f(hdp) or f(ndp) produces a new YD (Dust) pixel or a new ND 
(Non Dust) pixel respectively, when the thresholds p(Hdp) and p(Ndp) arc eventually 
exceeded as illustrated in Fig 1 3(c). 

Figure 2(b) describes an alternate embodiment of the dust map creation process. 
20 Block 1210 describes the preparation of the dust map which includes either opening an 
existing map (1216) or creating a new one (1214), where all pixels as a starting point are non 
dust or WHITE. After the map is correlated to the shooting conditions, 300, a dust map I-dM 
is created in block 1220. All pixels in the image (1222) receive the following values (1224) 
based on the luminance value of the pixel: 

25 Case: 

If luminance is less than DARKJTHRESHOLD, then I-dM (X,Y) = MAY8E _DUST; 
If luminance is greater than WHITE_THRESHLOD, then l-dM(x,y) = WHITE PIXEL; 
OTHERWISE l-dM(x,y) = D0NT_KN0W. 

Once all pixels are analyzed (1220) they are then clustered, (1296) into dust regions 
30 or dust specs in a I-pM dust map. The next step is to create a dust map dM (1240) which is 
continuous based on the value of the individual pixels. In the final stage (1250), the dust map 
is separated into regions based on at least one predetermined value THRESHOLD to create at 
least one binary mask. 
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An additional illustrative example of how the alternative dust map creation/updating 
process of Fig 2(b) can create or remove dust specs is given in Figs I4(a)-(f). It is assumed 
that before the current image is loaded, the pM and dM maps have the aspect depicted in 
diagrams 14(a) and 14(b) respectively. Note that pM is a binary map, whereas dM is a multi- 
5 level (continuous) dust map. After the current image goes through the 1220 block of Fig 2(b), 
the resulting I-dM has the aspect illustrated in Fig 14(c) - a three-level, or trinary, map. After 
the processing in the 1296 block of Fig 2(b), the I-pM map depicted in Fig 14(d) is obtained - 
also a trinary map. Next, for all the pixels belonging to "MAYBE DUST* regions, the values 
are decremented by a statistical quanta "quanta" as specified in block 1240 of Fig 2(b). The 
10 aspect of the resulting dM' map is similar to that of Fig 14(b), except that the actual values of 
the "MAYBE DUST" pixels which are, consequently, smaller. Now a thresholding dM' 
(block 1250 of Fig 2(b)) the new pM' map is obtained as illustrated in Fig 14(f). As can be 
seen from 14(f), a new dust pixel will have appeared in the left dust spot and a pixel will have 
dissapeared in the right dust spot of the binary map. (Note that for the purposes of this 
15 example it has been assumed that one pixel in each of the dust blocks of Fig 14(a) is close 
enough to the DUST/NOT-DUST thresholds to be altered by applying the single process step 
illustrated in Figs 14(a)-(f). In practical embodiments it will normally take multiple process 
steps for pixels to transition from DUST or NOT-DUST.} 

Figure 3 illustrates the process of correlating the image to a default settings of the dust 
20 map. This process defines correlating the image to the dust map, the dust map to a new dust 
map or the dust map to the image. Said correlation processes are interchangeable and can be 
done by adjusting the image to the map or adjusting the map to the image. In some cases 
both acquired image and dust map may be adjusted to some common ground as was 
previously described.. 

25 To begin with, the dust map is being loaded (112) and the default data on which the 

map was generated is extracted (310). Such data may include the lens type, the aperture and 
the focal length associated wit the default state. In concurrence, the information from the 
acquired image (304) is extracted (320) and compared to the one of the dust map. 

As explained in the mathematical model of the optical system, the two principle 

30 adjustments between the dust map and the image are based on focal length, and on aperture, 
each creating a different artifact that should be addressed. Knowledge of the phenomena may 
assist in creating a better detection and correction of the dust artifact. Alternatively, in a 
separate embodiment, analysis of the image and the modification of the dust as aperture and 
focal length are changed, may be used to construct an empirical model that describes 
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transformations that define changes to the dust as a function of changes in the^lens type, the 
focal length and the aperture. This latter approach is particularly useful in the determination 
of non-distortions which may occur for certain lens geometry's. 

The mathematical distortion of the dust as a function of the aperture is illustrated in 
5 figures 7A-7E. The geometrical optics illustrations of the above phenomena are depicted in 
Figures 6A-6F. Referring to figure 3, after extracting the data, the following step is 
modification of the map and or the image based on focal length (900), and based on aperture 
(800). The following steps are further defined in Figure 9 and Figure 8 respectively. 
Following these two steps (800 and 900) the image and the Dust Map are considered to be 
10 correlated. The correlated map cM is no longer binary because it defines both the shift and 
the fall off which is continuous. Figure 4A (400) defines the process of detecting and 
removing the dust from the image. The input which is the image I is loaded, if it is not 
already in memory (404) and the correlated dust map is cM is loaded (402) if already not in 
memory. 

15 The process of detecting and removing the dust is done per each dust spec. This 

process is highly parallelized and can be performed as a single path over the image, or in 
strips. As defined and justified by the physical phenomena, the method of correcting dust 
artifacts is defined based on two different operations. The first is the retouching or the in- 
painting of regions with no data (440), or data that is close to noise which must be recreated 

20 as defined in 430 and later on in Figure 7D. The second of said correction operations is based 
on the tapering degradation of surrounding pixels as a function of the aperture, as illustrated 
in Figures 7B - 7D. 

Referring to the image enhancement portions, where the data exists but most likely not in its 
full form, due to the fact that some of the dust affects the quality of the image, but some 
25 regions still contain some data (420), an image enhancement algorithm is performed on the 
pixels (430). In a simplified embodiment (432), assuming that the optical model is simplified 
to a degradation in the overall brightness as defined by the OPACITY, the enhanced pixels 
will receive a value inversely related to the OPACITY i.e. (432) 

/•(x ( v)=-M_ 
OPACITY 

30 To maintain a check of the validity of the model, the pixels before being operated on 

may be send for validation (500) as described in Figure 5. The second portion of the image 
correction is the restoration or the in-painting (450). In this case, the area behind the dust has 
no relevant data to enhance, or if there is, this data is relatively close to the overall noise level 
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and thus can not be enhanced. Therefore, it is necessary to in-paint every pixel based on an 
analysis of the surrounding region to the dust (470). In the case where the enhancement 
regions as defined in block 430 are of good quality, those pixels as well may contribute to the 
in-painting. 

5 Figure 4B illustrates a certain embodiment of the in-painting process. In general, each 

pixel in the obscured region, 480, is to be filled, 482, based on its surrounding pixels. In this 
specific embodiment, the pixels are filled in based on information of the surrounding non- 
affected pixels. This specific algorithm takes into account that pixels closer to the periphery 
have a better chance to be anticipated by the external pixels. Therefore the in-painting 
10 proceeds from the outside inwards, typically on a spiral path. 

This process of spiraling inwards is also depicted in Figure 4C. In it, given a dust 
spec J 400 in a grid 1410 the dust is taken a digital form, of the surrounding bounding area 
1420. External pixels such as 1430 are not part of the dust region, while internal ones such as 
1446, 1448 or 1450 are. 

15 An algorithm that works on the periphery moving inwards, as defined in Figure 4B 

blocks 470 is depicted in Figure 4D as follows: In the first step all peripheral pixels, 
numbered from 1 to 20 are being operated on as defined in Figure 4B, block 474. After that, 
all of the above mentioned twenty pixels, as defined in Figure 4F Block 1460 are removed, 
according to block 476 of Figure 4B, from the region leaving a smaller dust spec of shape 
20 1470. This modified dust spec, has a different set of peripheral pixels numbered 21-33. After 
removing those pixels by the in painting process, block 476, a smaller dust kernel as depicted 
in 1480 is left with only three pixels 34, 35 and 36. The process of filling in the pixels need 
not be spiral. In a different alternative, the process follows the pattern of the region 
surrounding the dust. For example lines and edges, or other high frequency information are 
25 being filled in prior to filling in the smooth regions. This approach will prevent unnecessary 
blurring due to the in-painting process. The criteria for locating pixels containing such high 
frequency data can be based on a determination of the overall gradient around a pixel, 462. 
The justification is that in the case of a steep edge, it is not advisable to copy information for 
the surrounding pixels with the edge value. By doing so, the in painting process will maintain 
30 any high frequency information in the picture such as lines and edges. 

An illustration of this is given in figures 4C, 4D and 4E. Referring to figure 4C, the 
same dust 1400 as in figure 4F with the bounding box 1420 on a grid 1410 is obscuring a 
picture including high frequency data such as the letter A, 1440. Take a small section based 
on 9 pixels as depicted in figure 4D where pixel 1490 is surrounded by 8 pixels 1491, 1492, 
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1493, 1494, 1495, 1496, 1497, 1498. In a simplification, each pixel receives the average 
value between the black and white regions. This is depicted in figure. 4e. In this case, the 
pixels 1490 and its surrounding 8 pixels 1491, 1492, 1493, 1494, 1495, 1496, 1497, 1498 
have a digital value of 210, 220, 48, 35, 145, 180, 253, 245 and 250 respectively. It is clear 
that the best value of pixel 1490 will come form its surrounding pixels with the smallest 
gradient. Because in practice the value of pixel 1490 can not be determined, but is the center 
of the in painting, the value of the gradient will be based on the value of the gradients of the 
pixels around it and extrapolated to this pixel. In this specific example, the differential 
horizontal gradient between pixels 1497 and 1493, is the largest while the vertical gradient of 
1495 and 1491 will most likely be the same. Therefore, a preferred value of 210 will be based 
on the extrapolated average of the two gradients of its top and bottom pixels. 

Figure 5 describes a tool that may be used in various stages of the process,, namely 
quality control. This process can be called from the main workflow (Figure 1) the creation of 
the dust map (Figures 2A and 2B) or the image restoration and enhancement (Figure 4A and 
4B respectively). The purpose of this tool is to perform the necessary quality check to assure 
that pixels will not be wrongly classified, nor mistakenly corrected. 

As examples of the principle categories of tests that may be performed as part of the 

quality control process we cite: 

(i) in a small aperture the region behind the dust should be close to if not totally 
obscured; if this is not the case (530 NO), then the dust map may need to be 
reviewed (550); or the image may not be correctly correlated to the dust map. 
(This can happen when the image is from a different time, or from a different 
acquisition device. In this case, the software will create a new dust map (200) to 
correspond to the new dust or lack thereof, or inform the user that it can not 
correct this image.) 

(ii) In the case where there is not enough information (530 MAYBE), there is no 
conclusive evidence to reject the correlation and thus the process continues (580) 
with no corrective action. 

(iii) In the case that the image displays information that may concur with the dust 
map, the software may continue (580) or prior to that, enhance the likelihood in 
the dust map (200) that the region is indeed dust. 

Figure 8 describes the adjustment of the dust map to the image acquisition parameters 
based on the aperture. Simplifying, the more closed the aperture is, the crisper and more 
noticeable the dust is. In other words, for example images taken with an f-stop of f-32, the 
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dust will be very prominent and opaque, while the same image taken at f-2.8 may display no 
visual degradation of the image. 

The correction of the image should take that information into account, to prevent over- 
correction of the dust artifact. 
5 The acquisition information and the corresponding dust map default setup are 

extracted in blocks 326 and 312 respectively. Then, for each dust spec in the map 810, the 
size of the region that is still obscured by the dust is calculated, as defined by the 
mathematical model. In some cases, when the aperture is very open, this region may decline 
to 0. In others, where the aperture is almost fully closed, the size may be close to the size of 
10 the dust. Alternatively, this step, 820, may be implemented as part of a preparation step, and 
recorded in a database, or LUT. 

The process then splits in two. The fully obscured region will be marked in 834 pixel 
by pixel 832 in a loop 834, 835 and will be treated by the in-painting process as defined in 
Figure 4B. A semi opaque dust map, is created in the loop 840, 841 for each pixel. Each of 
15 the pixels 842, is assigned an OPACITY value 844, based on the mathematical model as 
described previously in Figures 7A-7E. A dust spec that is only partially attenuated will go 
through a inverse filtering of the image data within a region of the image corresponding to 
said dust spec, as described in Figure 4 block 430, with a specific embodiment in block 432. 
This inverse filtering process may take into account the signal to noise ratio to avoid 
20 enhancing data which is not part of the original image. For example, the region around the 
dust may have an over-shoot similar to a high pass filter, which may manifest itself in the 
form of an aura around the dust. This aura should to be taken into account before enhancing 
the regions. 

Figure 9 describes the adjustment of the Dust Map based on the Focal length, and the 
25 specific lens. The scientific background is explained in figures 6F.7F. As described before, 
the spatial shift of a dust spec as a function of focal length for a specific lens is a function of 
(equation 1 1) the thickness of the window- tw, which is constant for a given camera, and exit 
pupil position, Pe, which varies based on the lens system and a variable focal length in the 
case of case of a zoom lens. Given an image and a dust map, the pertinent information is 
30 loaded, as described in Figure 3, namely the focal lens and lens type of the camera, 326, the 
focal length and lens type in the dust map 312 and the camera distance of dust to the sensor 
318. The process then goes through all known dust specs in the map 910 and calculates the 
shift of the dust as explained in figure 6E, 6F. The coordinates of the center pixel of a dust 
spec are calculated from the center of the optical path, 922, and the corresponding shift of the 
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equivalent dust region on the image is calculated 924. Moreover, because the shift is a 
function of the location (x,y) of the dust the dust shape may also need to be transformed. 

In some cases, it is impossible to get the data on the exit pupil, nor the distance the 
dust is from the sensor. Such cases may be when the application has no a-priori knowledge of 
the camera or the lens that was used. It is still possible in those cases to create a reasonable 
dust map, by empirically reconstructing the parameter based on analysis of the images, the 
shift in the dust and the falloff of the dust edges. Such techniques are defined in Figure 10 
and 1 1 defining the process of estimating, based on empirical data, the values of the 
parameters in the formulae that defines the change in dust as a function of a change in 
aperture. Figure 10 defines the process of creating the mathematical formulae based 
on empirical data to parameterize the change in dust as a function of change in focal length. 
In general, a proposed embodiment relies on the fact that when a dust is found, a pattern 

matching can be applied to find the shift in the dust. Based on this information, p can be 

calculated as defined in Equation 1 1-a. If K is known then Pe can be calculated as recited in 
equation 1 1-b. Specifically, in a preferred embodiment, an image is acquired, 1010 and a dust 
map is calculated 1012. A second image is captured, 1030 with a different focal length than 
the first image, and a dust map is calculated 1012. The process repeatedly tries to find two 
dust spots in the two dust maps 1040. If no dust specs are correlated the process is repeated 
for consecutive images, 1020. The process of finding dust specs is calculated by applying a 
local correlation to each of the dust specs. Preferably, based on Equation 1 1-a, the further the 
dust is from the center, the better the precision is. When two specs arc determined to be from 
the same dust spec the disparity between the specs is calculated 1042. The ratio between the 

shifted pixels is calculated. This ratio is the numerical empirical estimation of 'iv/Pg in 
equation 1 1-a. Moreover, if the distance of the dust to the sensor is known, the Exit pupil of 
the lens can be calculated based on the same equation. 

Figure 1 1 defines the process of estimating based on empirical data the values of 
the parameters in the formulae that defines the change in dust as a function of change in 
aperture The process is similar to the one described for estimation of the focal length, albeit 
the fact that the parameters calculated are different. Specifically, a first image is acquired, 
1 1 10 and dust is detected, 1 120 in this image. If the image is not appropriate for detecting the 
dust, or if the probability is low for the dust regions, this image is rejected for this purpose 
and another image is chosen. For the empirical comparison, a second image is captured 



I 140, or alternatively a set of images 1130, all with varying aperture, to enable acceptable 
sampling set. The process then looks for a detected dust region with high level of probability. 
In the case that all other parameters are similar except the aperture, the process can search for 
dust regions in the same coordinates that the original image dust regions were found. The 
5 dust regions of the two or more images are correlated, 1 160. The process continues for a 
sufficient amount of dust regions, 1 168, which in some cases can even be a single one, and 
sufficient amount of images, 1 169, which can also be, depending on the confidence level, a 
single image. Once the dust regions are correlated 1160, the fall off due to the change of 
aperture is calculated, 1 172, on a pixel by pixel basis, 11 70, for every pixel in a dust region, 
10 1179. Based on this information, the fall off function is calculated. 1180. In a preferred 
embodiment, the fall off function is determined as a function of the distance of a pixel from 
the periphery of the dust, as well as the aperture. 

Alternatively, the dust specs may also be determined by trying to correlate the dust 
spec in the map to the acquired image. Such correlation can be performed within a reasonable 
15 window size or even on the diagonal line between the dust and the center of the optical path, 
in the direction that the dust may move. By gaining knowledge on the movement of a single 
dust spec, as explained in formulae 1 1-a, all other dust specs shift can be determined. It is 
also possible to determine whether the camera should be physically cleaned based on analysis 
of the dust in the camera and the specific dust patterns. This is illustrated in the flowchart of 
20 figure 12. An example will be that certain width of dust will not allow correct in-painting 
based on the surround. Another example will be the overall number of dust specs or the 
overall relative area that the dust covers. The input for such analysis is a dust map, 1200. 
This map can be similar to the dust map generated in block 200, or any other representation 
of a dust map or a calibration map, generated automatically or manually by the photographer. 
25 Such analysis need not be performed for every image. A process, 1260 determines whether to 
perform such analysis. Examples to trigger this process are the time since the last analysis, 
change of lenses, which may create the introduction of dust, or a message from the quality 
control analysis of the dust map, as defined in Figure 5 block 500, that the dust has shifted or 
that the dust no longer corresponds to the map. In general, any changes in the dust structure 
30 may be a justification to trigger the analysis process. When no analysis is desired, 1262, the 
process terminates. Otherwise, the analysis is performed, 1270. The analysis is performed for 
each dust spec individually, 1272, and then the results are accumulated for the whole image. 
For each dust region, 1273, some parameters are extracted, including, but not limited to the 
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area of a dust region; maximum width of a dust region; the distance between dust regions 
and neighboring regions; the relative movement of dust regions from the last analysis; the 
occurrence of new dust specs since the last analysis, etc. Following this analysis, the results 
are statistically analyzed and summarized to include such data as - the nmber of dust specs, 
the area of dust, in size and in percentage for the entire image; the largest width of dust; the 
largest area of a dust spec; changes in area of dust since last analysis and changes of dust 
particles since last analysis. In an alternate embodiment, either automatically or based on the 
photographers preference, this analysis may be displayed or saved in a log file for future 
reference, 1221. Each of the aforementioned criteria may have at least one acceptable 
threshold value. This threshold value is determined empirically by the manufacturer in terms 
of the maximum acceptable dust that can be corrected in software. Alternatively, this data 
may be adjusted to by the photographer based on their tolerance level for dust artifacts. If 
any of the parameters exceeds an acceptable threshold, the user is informed, 1290, that the 
camera should be manually maintained and cleaned. 

Alternatively, 1285, this same process may be used as a tool to inform the user of 
changes in the dust. Such information is particularly important in the case that the dust 
correction algorithm is based on the creation of a calibration image. In this case, the analysis 
will be used to inform the user that a new calibration image should be acquired to support the 
correction of dust in future images. 

Alternatively, the process of analysis described above may also be incorporated in a 
maintenance procedure, where the camera, after being cleaned up will perform an auto-test, 
as described in Figure 12, to verify that the camera now is indeed clean. In such cases, the 
threshold parameters are of course substantially more restrictive and demanding, to assure 
high quality maintenance. 

There are many alternatives to the preferred embodiments described above that may 
be incorporated into a image processing method, a digital camera, and/or an image processing 
system including a digital camera and an external image processing device that may allow the 
present invention to be practiced advantageously in certain embodiments. In a camera 
application, the unique location of the actual dust relative to the object and to the image plane 
provide information about extracted parameter-dependent characteristics of dust artifact in 
the images. It is possible to make determinations as to where the dust actually is in the system 
by analyzing multiple images taken with different extracted parameters, e.g., on the sensor 
window, or in an image of an original object which itself is being imaged such as in film 
imaging. Techniques may be also employed involving correcting for dust defects based on 



30 

the geometry of said dust or of the camera. Further techniques may involve utilizing camera 
metadata to enhance the detection and correction processes for defects. A method may 
indude analyzing the images in comparison to a predetermined dust map to establish the 
validity of the dust over progressions of time. The method may further involve mapping the 
5 acquired image to a predetermined default acquisition condition as a function of the lens type 
and the focal length that was used at acquisition. 

A method may further include mapping a dust spec as depicted in the dust map and 
the suspected dust specs in the acquired image based on a calculated transformation of the 
dust as a function of the lens and the aperture, or other extracted parameters used to acquire 
10 the image. A dust detection and/or correction technique may be applied post priori to a 
collection of images, or individually to images as they are added to a collection. The map 
may be generated a priori to the introduction of an image, or dynamically and in concurrence 
to the introduction of new images. The method may further include steps of providing a 
statistical confidence level as to the fact that a certain region is indeed part of a dust spec. 
J 5 The method may further provide tools to determine whether the acquisition device may 
benefit from some maintenance. 

A method of automatically determining whether to recommend servicing a digital 
image acquisition system including a digital camera based on dust analysis may be 
advantageously employed. In addition, in methods that may be performed according to 
20 preferred embodiments herein, the operations have been described in selected typographical 
sequences. However, the sequences have been selected and so ordered for typographical 
convenience and are not intended to imply any particular order for performing the operations, 
unless a particular ordering is expressly provided or understood by those skilled in the art as 
being necessary. 
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Claims: 

I . A method of automatically correcting dust artifact regions within images acquired by a 
digital acquisition device including an optical system, comprising: 

(a) digitally-acquiring one or more original images with said digital acquisition 
device; 

(b) determining probabilities that certain pixels correspond to dust artifact regions 
within said one or more digitally-acquired images; 

(c) associating the dust artifact regions with one or more extracted parameters 
relating to the optical system when the one or more images were acquired; 

(d) forming a statistical dust map including mapped dust regions based on the dust 
artifact probability determining and associating; 

(e) correcting pixels corresponding to dust artifact regions within each of said one or 
more original images based on the associated statistical dust map. 

2. The method of claim 1, said one or more extracted parameters comprising aperture 
size, F-number, magnification, lens type or focal length of an optical system of the digital 
acquisition device, or combinations thereof. 

3. The method of claim 2, said one or more extracted parameters are calculated 
empirically from comparison of one or more said dust artifact regions within said multiple 
original digital images with said digital acquisition device. 

4. The method of claim 2, said one or more extracted parameters comprising aperture size 
or focal length or both. 

5. The method of claim 1, further comprising digitally-acquiring further images with said 
digital acquisition device, repeating said determining and associating, and updating said 
statistical dust map including updating said mapped dust regions based on the further dust 
artifact determining and associating. 

6. The method of claim 5, further comprising correcting pixels corresponding to 
correlated dust artifact regions within said further images based on the updated, associated 
statistical dust map. 

7. The method of claim 5, further comprising updating one or more of said one or more 
original images based on said updating of said associated statistical dust map. 
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8. The method of claim 5, further comprising repeating for said further digitally-acquired 
images said determining and associating, and updating said statistical dust map including 
updating said mapped dust regions based on the additional dust artifact determining and 
associating. 

5 9. The method of claim 5, further comprising limiting updating one or more of said 
further and original images based on said updating of said associated statistical dust map to 
updates that do not include appearance of new dust or movement of existing dust. 

10. The method of claim 5, further comprising limiting updating one or more of said 
further and original images based on said updating of said associated statistical dust map to 

10 updates that include previously determined dust artifact regions. 

11. The method of claim 5 further comprising creating a version description of changes in 
said statistical dust map. 

12. The method of claim 1 1, further comprising updating one or more of said further and 
original images based on said updating of said associated statistical dust map based on said 

15 version description. 

13. The method of claim 5, wherein said version is based on a chronological time stamp. 

14. The method of claim 5, wherein said version is based on replacement of a lens. 

15. The method of claim 5, wherein said version information comprises change of said 
probabilities in said statistical dust map that certain pixels correspond to dust artifact regions. 

20 16. The method of claim 5, wherein said version information includes one or more 

parameters comprising of change in dust location, change in dust position, appearance of new 
dust region, disappearance of existing dust region. 

17. The method of claim 5, wherein further comprising determining whether dust map 
needs to be replaced based on determining that sufficient disparity exists based amount and 

25 quality of said changes in said statistical dust map. 

18. The method of claim 1, said image correction method being automatically performed 
within a digital camera that comprises said optical system, said sensor array, said processing 
electronics and said memory. 

19. The method of claim 1, said image correction method being performed at least in part 
30 within an external processing device that couples with a digital camera that comprises said 
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optical system and said sensor array to form a digital image acquisition and processing 
system that also comprises said processing electronics and said memory. 

20. The method of claim 1 9, the programming instructions being stored on a memory 
within the external device which performs the image correction method. 

21. The method of claim 1, said determining comprising determining probabilities that 
certain pixels correspond to dust artifact regions within said acquired images based at least in 
part on a comparison of suspected dust artifact regions within two or more digitally-acquired 
images, or on a pixel analysis of the suspected dust artifact regions in view of predetermined 
characteristics indicative of the presence of a dust artifact region, or both. 

22. The method of claim 2], further comprising eliminating certain suspected dust 
artifact regions as having a probability below a first threshold value. 

23. The method of claim 22, further comprising judging certain further dust artifact 
regions as having a probability above said threshold value, such as to be subject to further 
probability determining including comparison with further acquired images prior to judging 
whether each said further dust artifact region will be subject to said eliminating operation. 

24. The method of claim 22, further comprising judging certain probable dust artifact 
regions as having a probability above a second threshold value such as to be subject to said 
correcting operation. 

25. The method of claim 24, wherein said first and second threshold values are different. 

26. The method of claim 25, further comprising judging certain further dust artifact 
regions as having a probability between said first and said second threshold values, such as to 
be subject to further probability determining including comparison with further acquired 
images prior to judging whether each said further dust artifact region will be subject to said 
correcting operation. 

27. The method of claim 21, further comprising judging certain probable dust artifact 
regions as having a probability above a threshold value such as to be subject to said 
correcting operation. 

28. The method of claim 27, further comprising judging certain further dust artifact 
regions as having a probability below said threshold value, such as to be subject to further 
probability determining including comparison with further acquired images prior to judging 
whether each said further dust artifact region will be subject to said correcting operation. 



29. The method of claim 21, wherein said probability determining includes weighting 
suspected dust artifact regions according to one or more predetermined probability weighting 
assessment conditions. 

30. The method of cJaim 29, said one or more weighting assessment conditions 
5 comprising size, shape, brightness or opacity of said suspected dust artifact regions, or degree 

of similarity in size, shape, brightness, opacity or location with one or more suspected dust 
artifact regions in one or more other images, or combinations thereof. 

31. The method of cJaim 1, wherein said determining is based at least in part on a 
comparison of suspected dust artifact regions within two or more digitally-acquired images. 

10 32. The method of claim 1, wherein said determining of said probabilities is further based 
on a pixel analysis of the suspected dust artifact regions in view of predetermined 
characteristics indicative of the presence of a dust artifact region. 

33. The method of claim 1, wherein said suspected dust artifact regions of said at least 
two images comprise inner regions and aura regions, and wherein said correcting comprises a 

15 first correcting of said aura regions and a second correcting of said inner regions. 

34. The method of claim 1, the dust map including dust artifact regions with probabilities 
above a threshold probability and not including regions with lower probabilities. 

35. The method of claim 1, the dust map including dust artifact regions with probabilities 
above a first threshold value, not including regions with probabilities below a second 

20 threshold value, and where further regions having a probability between said first and second 
threshold values exist within said dust map, then further image information is combined into 
said probability analysis before determining whether said region are included as dust artifact 
regions. 

36. The method of claim 1 wherein said determining probabilities further comprises 
25 statistically combining a plurality of individual probabilities based on each said regions 

within two or more said images. 

37. The method of claim 1, further comprising determining probabilities that certain 
pixels correspond to regions free of dust within said images based at least in part on a 
comparison of suspected dust artifact regions within one or more of said images. 

30 38. The method of claim 37, further comprising eliminating certain suspected dust 
artifact based on probabilities that certain pixels correspond to regions free of dust 
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39. The method of claim I, further comprising validating whether said further digitally- 
acquired image has non contradicting probability data that certain pixels correspond to dust 
artifact regions within said further digitally-acquired image prior to correcting pixels 
corresponding to correlated dust artifact regions within further digitally-acquired images 
based on the associated statistical dust map. 

40. The method of claim 1, wherein said suspected dust artifact regions of said further 
digitally images comprise shadow regions and aura regions. 

41. The method of claim 40, said focal length extracted parameters are calculated 
empirically from comparison of the transposition of said shadow regions of said dust artifact 
regions within said multiple original digital images with said digital acquisition device. 

42. The method of claim 40, said aperture extracted parameters are calculated 
empirically from comparison of the fall off of said aura regions of said dust artifact regions 
within said multiple original digital images with said digital acquisition device. 

43. The method as recited in claim 40, wherein said correcting comprises a first 
correcting of said aura regions and a second correction of said shadow regions. 

44. The method of claim 40, said determining with respect to a shadow region being 
based on an extracted parameter-dependent shadow region analysis, wherein the shadow 
region analysis presumes that certain regions on a sensor of the digital image acquisition 
device are fully obscured by said dust. 

45. The method of claim 44, wherein the shadow region analysis includes calculating 
effects of differences in values of the one or more extracted parameters in different images on 
dust artifact illumination, shape, position, reflection or transmission properties, distance of 
dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

46. The method of claim 45, wherein said different images are acquired with different 
values of said one or more extracted parameters. 

47. The method of claim 45, wherein said different images are acquired of different 
objects. 

48. The method of claim 45, wherein said different images are acquired of different 
scene. 

49. The method of claim 40, said determining with respect to an aura region being based 
on an extracted parameter-dependent aura region analysis, wherein the aura region analysis 
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presumes that certain regions on a of the digital image acquisition device are partially 
obscured by said dust. 

50. The method of claim 49, wherein said aura region analysis includes calculating effects 
of differences in values of the one or more extracted parameters in different images on dust 

5 artifact illumination, shape, position, reflection or transmission properties, distance of dust to 
the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

51. The method of claim 50, wherein said different images are acquired with different 
values of said one or more extracted parameters. 

52. The method of claim 50, wherein said different images are acquired of different 
10 objects. 

53. The method of claim 40, the correcting operation comprising in-painting or 
restoration, or both. 

54. The method of claim 53, said correcting including in-painting the shadow region. 

55. The method of claim 54, said in-painting including determining and applying shadow 
J 5 region correction spectral information based on spectral information obtained from pixels 

outside said shadow region. 

56. The method of claim 53, said correcting including restoration of the aura region. 

57. The method of claim 56, said restoration including determining and applying aura 
region correction spectral information based on spectral information obtained from pixels 

20 within said aura region. 

58. The method of claim 1, the method being performed on raw image data as captured 
by a camera sensor. 

59. The method of claim 1, said image correction method being performed on a processed 
image after being converted from raw format to a known red, green, blue representation. 

25 60. The method of claim 1, wherein said correcting includes replacing said pixels within 
said one or more digitally-acquired images with new pixels. 

61. The method of claim I, wherein said correcting includes enhancing said values of 
pixels within said one or more digitally-acquired images. 

62. The method of claim 1, wherein correcting instructions are kept in an external 
30 location to the image data. 
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63. The method of claim I, said external location comprising an image header. j 

i 

64. The method of claim I, the dust artifact determining operation including: j 

(a) loading the statistical dust map; 

(b) loading extracted parameter information of a present image; 

5 (c) performing calculations within the statistical dust map having extracted 

parameter variable-dependencies; and 

(d) comparing dust artifact detection data with the extracted parameter dependent 
statistical dust map data. 

65. The method of claim 64, the extracted parameter information including values of 
10 aperture size and focal length. 

66. The method of claim 65, the extracted parameter information further including lens 
type information. 

67. The method of claim 1, the dust artifact determining operation including: 

(a) loading the statistical dust map; 
]5 (b) loading extracted parameter information of a present image; 

(c) performing a calculation for relating the statistical dust map with the present 
image according to a selected value of at least one extracted parameter which 
is otherwise uncorrected between the present image and the dust map; and 

(d) comparing dust artifact detection data with the now correlated statistical dust 
20 map data. 

68. The method of claim 1, wherein said suspected dust artifact regions of said at least 
two images comprise inner regions and aura regions, and wherein said comparison comprises 
a first comparison of said inner regions and a second comparison of said aura regions. 

69. The method of claim 68, said dust artifact regions including an aura region partially 
25 obscured by dust and a shadow region substantially obscured by dust inside said aura region. 

70. The method of claim 69, said determining with respect to a shadow region being 
based on an extracted parameter-dependent shadow region analysis, wherein the shadow 
region analysis presumes that certain regions on a sensor of the digital image acquisition 
device are fully obscured by said dust. 

I 

I" 

! 

i 
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71. The method of claim 70, wherein the shadow region analysis includes calculating 
effects of differences in values of the one or more extracted parameters in different images on 
dust artifact illumination, shape, position, reflection or transmission properties, distance of 
dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

5 72. The method of claim 71, said different images having been acquired of different 

objects. 

73. The method of claim 71, wherein said different images are acquired of different 
scene. 

74. The method of claim 69, said determining with respect to an aura region being based 
10 on an extracted parameter-dependent aura region analysis, wherein the aura region analysis 

presumes that certain regions on a of the digital image acquisition device arc partially 
obscured by said dust. 

75. The method of claim 74, wherein said aura region analysis includes calculating 
effects of differences in values of the one or more extracted parameters in different images on 

15 dust artifact illumination, shape, position, reflection or transmission properties, distance of 
dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

76. The method of claim 75, said different images having been acquired of different 
objects. 

77. The method of claim 69, the correcting operation comprising in-painting or 
20 restoration, or both. 

78. The method of claim 77, said correcting including in-painting the shadow region. 

79. The method of claim 78, said in-painting including determining and applying shadow 
region correction spectral information based on spectral information obtained from pixels 
outside said shadow region. 

25 80. The method of claim 77, said correcting including restoration of the aura region. 

81. The method of claim 80, said restoration including determining and applying aura 
region correction spectral information based on spectral information obtained from pixels 
within said aura region. 

82. The method of claim I, further comprising sorting dust artifact distribution data 
30 within said dust map according to meta-data. 
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83. The method of claim 67, the extracted parameter information including values of 
aperture size and focal length. 

84. The method of claim 67, the extracted parameter information further including lens 
type information. 

85. The method of claim 67, the extracted parameters comprising focal length, 
magnification or type of lens assembly, or aperture size or position, or combinations thereof. 

86. The method of claim 67, the extracted parameters further comprising distance of the 
actual dust object from one or more positions in plane of the electronic sensor array. 

87. The method of claim 67, the method further comprising determining that sufficient 
disparity exists between dust artifact determinations within sequential images to determine 
that the digital acquisition device may have been cleaned and that the dust artifact 
determinations prior to the cleaning will not be used in the dust map. 

88. The method of claim 67, the correcting comprising in-painting or restoration, or both. 

89. The method of cJaim 88, the in-painting operation being performed on pixels 
determined to have substantially no image-relevant spectral information, or insufficient 
image-relevant spectral information, associated with their corresponding data. 

90. The method of claim 89, the in-painting including creation of new pixel values based 
on characteristics of surrounding pixels to the dust region. 

91. The method of claim 90, one or more characteristics of the surrounding pixels upon 
which said in-painting is based include the coJor, brightness, gradient, edge detection, noise, 
pattern, texture, geometry, or combinations thereof. 

92. The method of claim 88, the restoration being performed on pixels determined to 
have sufficient image-relevant spectral information associated with their corresponding data. 

93. The method of claim 92, the restoration including modifying brightness based on an 
inverse relationship to the opacity of the dust artifact or a brightness characteristic of the 
pixels within the image, or both. 

94. The method of claim 93, the restoration further including enhancing color based on a 
present color characteristic of pixels within the image. 

95. The method according to any of methods 1-94 the dust artifact determining and 
associating operations being repeated for further acquired images after forming said statistical 
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dust map; said statistical dust map being modified based on said further images including 
mathematically correlating determined and associated dust artifact regions of said further 
acquired images with dust artifact regions in said dust map including combining data 
associated with dust artifact regions within images acquired with different values of one or 
5 more extracted parameters; and correcting digital data corresponding to the correlated dust 
artifact regions within acquired images based on said modified statistical dust map. 

96. The method of claim 95, the modifying including varying probabilities based on 
combining new probability data with probability data within said original dust map. 

97. The method of claim 95, the modifying including eliminating a dust artifact region of 
10 said original dust map due to its probability falling below a threshold value between 0 and 1 

after combining new probability data with probability data within said original dust map. 

98. The method of claim 95, the modifying including inserting a new dust artifact region 
not present within said original dust map due to its having a probability, based on analysis of 
multiple images, that is now above a threshold value between 0 and 1 after combining new 

15 probability data with probability data within said original dust map. 

99. The method of claim 95, further comprising reapplying said correcting digital data for 
images previously acquired after said modifying statistical dust map showed a better said 
statistical probability to said one or more said detected dust regions. 

100. The method of claim 95, said dust artifact regions including an aura region partially 
20 obscured by dust and a shadow region substantially obscured by dust, the shadow region 

being inside said aura region. 

101. The method of claim 100, said determining with respect to a shadow region being 
based on an extracted parameter-dependent shadow region analysis, wherein the shadow 
region analysis presumes that certain regions on a sensor of the digital image acquisition 

25 device are fully obscured by said dust. 

102. The method of claim 101, wherein the shadow region analysis includes calculating 
effects of differences in values of the one or more extracted parameters in different images on 
dust artifact illumination, shape, position, reflection or transmission properties, distance of 
dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

30 103. The method of claim 100, said determining with respect to an aura region being 

based on an extracted parameter-dependent aura region analysis, wherein the aura region 
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analysis presumes that certain regions on a of the digital image acquisition device are 
partially obscured by said dust. 

104. The method of claim 103, wherein said aura region analysis includes calculating 
effects of differences in values of the one or more extracted parameters in different images on 
dust artifact illumination, shape, position, reflection or transmission properties, distance of 
dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof. 

105. The method of claim 100, the correcting operation comprising in-painting or 
restoration, or both. 

106. The method of claim 105, said correcting including in-painting the shadow region. 

107. The method of claim 106, said in-painting including determining and applying 
shadow region correction spectral information based on spectral information obtained from 
pixels outside said shadow region. 

108. The method of claim 105, said correcting including restoration of the aura region. 

109. The method of claim 108, said restoration including determining and applying aura 
region correction spectral information based on spectral information obtained from pixels 
within said aura region. 

110. The method of claim 100, the correcting operation including calculating said aura 
region and said shadow region. 

111. The method of claim 110, the correcting operation further including correcting a 
shadow region approximately based on the correcting of the aura region. 

112. The method of claim 95, the statistical dust map including extracted parameter 
dependent-variables such that dust artifact region effects in one image are predicted to differ 
based upon values of the variables within the statistical dust map. 

1 13. The method of claim 67, wherein said relating comprises performing at least one 
linear transformation on the assumption of circular symmetry of the present image and a 
predetermined mathematical relationship between said extracted parameter and the 
coefficients of said linear transformation. 

1 14. The method of claim 1 13 where said linear transformation comprises performing a 
spatial correlation of the dust artifact regions of the dust map with the present image. 
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115. The method of claim 114 comprising performing additional linear transformations 
relating to additional properties of the dust map or the dust artifact regions contained within 
said dust map. 

116. The method of claim 115 where said additional properties include one or more of 
5 dust map luminance, transparency, opacity, scale, dust artifact region size, luminance, 

transparancy, opacity, scale, and spatial location relative to image center. 

117. The method of any of claims 67 and 113-116, wherein said relating further 
comprises performing at least one non-linear transformation based, at least in part, on 
predetermined data stored in a LUT relating the selected value of the extracted parameter to 

10 the coefficients of said non-linear transformation. 

118. The method of claim 117 where said non-linear transformation corrects for non- 
circular symmetry of the present image. 

119. The method of claim 117 where said non-linear transformation corrects for non- 
linear spatial mappings arising from the optical properties of the imaging subsystem used to 

1 5 acquire the present image. 

120. The method of claim 1 17 where said non-linear transformation is an affine 
transformation. 

121. The method of claim 1 wherein the determination of probabilities that certain pixels 
correspond to dust artifact regions further comprises: 

20 (a) determining whether a threshold distribution of dust artifacts is present within 

one or more of said digital images; and 

(b) indicating a need for service when at least said threshold distribution is 
determined to be present. 

122. The method of claim 12], wherein said one or more acquired images comprise one 
25 or more calibration images. 

123. The method of claim 121, said threshold distribution being determined based upon 
an analysis of the ability of an automatic blemish correction module of said digital image 
acquisition system to reasonably correct such blemishes within said images. 

124. The method of claim 121, further comprising: 



WO 2005/041558 



rCi /Lrziiu4/u i» i v : j 



(a) determining probabilities of dust artifact regions corresponding to said pixels 
within said digitally-acquired image; 

(b) associating the dust artifact regions with one or more extracted parameters 
relating to the optical system when the image was acquired; 

5 (c) forming a statistical record including dust artifact regions based on the dust 

artifact determining and associating; and 

(d) determining said threshold distribution based on predetermined characteristics 
of said statistical record. 

125. The method of claim 124, size or shape, or both, of said dust artifact regions being 
10 included within said predetermined characteristics. 

1 26. The method of claim 1 24, said indicating comprising notifying a user, based on said 
determining whether said threshold distribution is present, that said digital acquisition device 
needs to be serviced. 

127. The method of claim 121 wherein said one ore more acquired images are acquired 
] 5 with specific acquisition setting comprising one or more of aperture, shutter speed, 

sensitivity, and subject matter. 

128. The method of claim 127, wherein said specific acquisition settings are automatically 
determined in a specific calibration mode on said digital image acquisition system. 

129. The method of claim 121, wherein said analyzing is based on defined time interval 
20 since last said analyzing. 

130. The method of claim 121, wherein said analyzing is based on defined in relations with 
change of lenses. 
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